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Background Health systems worldwide are facing shortages in 
health professional workforce. Several studies have demonstrated the 
direct correlation between the availability of health workers, cover- 
age of health services, and population health outcomes. To address 
this shortage, online eLearning is increasingly being adopted in 
health professionals' education. To inform policy— making, in online 
eLearning, we need to determine its effectiveness. 

Methods We performed a systematic review of the effectiveness of 
online eLearning through a comprehensive search of the major da- 
tabases for randomised controlled trials that compared online eL- 
earning to traditional learning or alternative learning methods. The 
search period was from January 2000 to August 2013. We included 
articles which primarily focused on students' knowledge, skills, sat- 
isfaction and attitudes toward eLearning and cost-effectiveness and 
adverse effects as secondary outcomes. Two reviewers independent- 
ly extracted data from the included studies. Due to significant het- 
erogeneity among the included studies, we presented our results as 
a narrative synthesis. 

Findings Fifty-nine studies, including 6750 students enrolled in 
medicine, dentistry, nursing, physical therapy and pharmacy stud- 
ies, met the inclusion criteria. Twelve of the 50 studies testing knowl- 
edge gains found significantly higher gains in the online eLearning 
intervention groups compared to traditional learning, whereas 27 
did not detect significant differences or found mixed results. Eleven 
studies did not test for differences. Six studies detected significantly 
higher skill gains in the online eLearning intervention groups, whilst 
3 other studies testing skill gains did not detect differences between 
groups and 1 study showed mixed results. Twelve studies tested stu- 
dents' attitudes, of which 8 studies showed no differences in attitudes 
or preferences for online eLearning. Students' satisfaction was mea- 
sured in 29 studies, 4 studies showed higher satisfaction for online 
eLearning and 20 studies showed no difference in satisfaction be- 
tween online eLearning and traditional learning. Risk of bias was 
high for several of the included studies. 

Conclusion The current evidence base suggests that online eLearn- 
ing is equivalent, possibly superior to traditional learning. These 
findings present a potential incentive for policy makers to cautious- 
ly encourage its adoption, while respecting the heterogeneity among 
the studies. 
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Health care workers are central to any health system; their 
activities are aimed at enhancing the health of the commu- 
nity nation or region in general. However, these profession- 
als are distributed unevenly across the globe; countries with 
lower relative need have the highest numbers of health 
workers, while those with the greatest burden of disease 
have a much smaller health workforce. The health worker 
crisis is at its worst in sub-Saharan Africa and Asia. Inci- 
dentally, countries in this region are facing a double burden 
of both infectious and non-communicable diseases [ 1 ] , 
and they lack the funds, technology, infrastructure and 
trained health workers needed to provide basic health care 
service [2] . At this juncture; the WHO estimates a shortage 
of 7.2 million doctors, nurses, midwives and other health 
care professionals worldwide [3] . Addressing this shortfall 
in health care professionals through training requires a sub- 
stantial investment. 

Meanwhile, the Internet and the development of informa- 
tion technologies have revitalized the exchange of informa- 
tion and training worldwide. Consequently, eLearning is 
used increasingly in medical and health professional edu- 
cation, to tackle the global shortage of health workers. eL- 
earning is "an approach to teaching and learning, repre- 
senting all or part of the educational model applied, that is 
based on the use of electronic media and devices as tools 
for improving access to training, communication and in- 
teraction and that facilitates the adoption of new ways of 
understanding and developing learning" [4] . It does not 
only differ from traditional learning (ie, face— to— face learn- 
ing that takes place in a classroom environment) in the me- 
dium by which learning is delivered [5] , but also affects the 
teaching and learning approaches used. eLearning can take 
the form of a full eLearning approach, which is entirely 
driven by technology, or be a mix of the traditional and 
fully computer-based methodologies (blended learning). 
Blended learning might be more suitable for health care 
training because of the need to combine hands-on skills- 
based training at a practical level as well as self-directed 
learning [6—10]. 

Lately, eLearning has started to make way into the devel- 
oping countries and is believed to have huge potential for 
governments struggling to meet a growing demand for ed- 
ucation while facing an escalating shortage of teachers [11]. 
However, in both the developing and non— developing 
countries, network connectivity and bandwidth availabil- 
ity are the key obstacles to the effective delivery of online 
eLearning [5,12]. Despite this, eLearning presents many 
opportunities for universities, including the reduction of 
the delivery costs [13] , increased scalability [14] , improved 
access and availability by removing geographical and tem- 
poral barriers and allowing access to relevant experts and 
novel curricula [15]. 



Online eLearning represents a further evolution of com- 
puter—assisted or computer based or offline eLearning and 
is an important tool in medical training and may transform 
the way medicine is taught [16] . In the recent years, near- 
ly all medical schools in the USA and Canada employ on- 
line course materials [17] . eLearning could be full or blend- 
ed, full eLearning can be distributed geographically and/or 
temporally, and communication between student and 
teacher is handled electronically. This manuscript focuses 
on online eLearning; systematic review of offline eLearning 
is published in a parallel article [18]. 

Online eLearning approaches varied widely in configura- 
tion (tutorial, asynchronous discussion, live conferencing, 
etc.), instructional methods (eg, practice exercises, cogni- 
tive interactivity) and presentation [17]. The majority of 
reviews of effectiveness of online eLearning included ob- 
servational studies with several methodological deficiencies 
[16,17,19—22] . This systematic review aims to evaluate the 
effectiveness of online eLearning from randomised con- 
trolled trials conducted among undergraduate health pro- 
fessionals, specifically looking at its impact on students' 
knowledge, skills, attitudes and satisfaction. 

METHODS 

We conducted a systematic review following the Cochrane 
methodology [23]. 

Search methods for identification of 
studies 

Electronic searches. We limited our electronic searches to 
records published on or after the year 2000 in order to 
highlight recent developments. We developed a search 
strategy for MEDLINE (OvidSP) using a combination of 
keywords and MeSH terms that captured the types of in- 
tervention and the types of participants under evaluation 
in this systematic review (Table 1). We adapted the search 
strategy for use in EMBASE (OvidSP), PsycINFO (Ovid 
SP), Cochrane Central Register of Controlled Trials (CEN- 
TRAL), Web of Science, and Educational Resources Infor- 
mation Center (ERIC) (ProQuest). Where available, we 
used validated methodological filters to limit our searches 
to Randomised Controlled Trials (RCTs) and cluster RCTs 
(cRCTs). We ran the searches in August 2013. 

Searching other resources. We checked reference lists of 
the included studies and systematic reviews of the literature 
identified by our electronic searches for additional studies. 

Inclusion criteria 

Types of studies and participants. We included studies 
published in any language on students of (i) undergradu- 
ate, health— related university degrees; or (ii) basic, health- 
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Table 1. Search strategy for use in MEDLINE (Ovid SP)* 
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related vocational training programmes. We defined un- 
dergraduate education or basic vocational training as any 
type of study leading to a qualification that: (i) is recognised 
by the relevant governmental or professional bodies of the 
country where the studies were conducted; and (ii) entitles 
the qualification-holder to apply for entry level positions 
in the health care workforce. For this reason, graduate 
medical education courses from the USA were included. 

We considered studies on candidates for and holders of the 
qualifications listed in the Health Field of Education and 
Training of the International Standard Classification of Ed- 



ucation (ISCED-F) [24] , except studies on students of tra- 
ditional and complementary medicine. We hence included 
students reading dental studies, medicine, nursing and 
midwifery, medical diagnostic and treatment technology, 
therapy and rehabilitation, or pharmacy. Medicine and 
dentistry were classified under the umbrella term allied 
health professions. 

Types of intervention. First, we conducted a systematic 
mapping of the types of technologies used by the included 
studies to deliver the learning materials, through which we 
identified 6 broad categories of eLearning interventions, 
based on the technologies employed: (1) Offline comput- 
er—based eLearning, (2) Online and local area network- 
based eLearning, (3) Psychomotor skills trainer, (4) Virtu- 
al reality environments, (5) Digital game-based learning 
and (6) mLearning. We allocated each included study to 
the category that fitted the study best. Please refer to On- 
line Supplementary Document for a definition of these 
categories. 

We only included studies in which online eLearning inter- 
ventions were used to deliver the learning content, studies 
were categorized as online eLearning if the delivery of the 
learning content was done through the internet or intranet 
connections. Only studies that compared online eLearning 
or blended eLearning methods to: (i) traditional learning; (ii) 
an alternative eLearning or blended learning method; or (iii) 
no intervention was eligible for inclusion. These studies 
could either be studies where eLearning was the sole means 
by which the intervention was delivered or where eLearning 
was part of a complex, multi— component intervention. 

Types of outcome measures. To be eligible for inclusion, 
studies had to report at least 1 of the following primary or 
secondary outcomes. 

Primary outcomes: (1) Students' knowledge, measured us- 
ing any validated or non-validated instrument (eg, pre— 
and post-test scores, grades, perceived knowledge survey 
scores); (2) Students' skills, measured using any validated 
or non-validated instrument (eg, pre- and post— test scores, 
time to perform a procedure, number of errors made whilst 
performing a procedure, perceived up-skilling); (3) Stu- 
dents' satisfaction and attitudes towards eLearning, mea- 
sured using any validated or non-validated instrument (eg, 
self-efficacy satisfaction, acceptability). 

Secondary outcomes: (1) Health economic properties of 
the interventions (eg, implementation cost, return on in- 
vestment); (2) Adverse and/or unintended effects of eL- 
earning (eg, potential feelings of depression and loneliness, 
dropout risks [25] and "computer anxiety" [26]). 

We only considered studies to have measured students' sat- 
isfaction and attitudes towards eLearning if they met all of 
the following criteria: (i) they compared the differences be- 
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tween intervention and control groups for these outcomes; 
(ii) the content of the survey questionnaires related to the 
teaching method (ie, eLearning method, blended learning, 
or traditional learning); and (iii) the adjectives used in the 
survey questionnaires accurately described attitudes and/ 
or satisfaction. 

Study selection and data collection 

The study selection process is summarised in the PRISMA 
flow diagram (Figure 1). In brief, we screened the titles 
and abstracts of the citations identified by our electronic 
and manual searches to identify potentially relevant stud- 
ies, of which we assessed the full— text report to ensure they 
meet the inclusion criteria we specified. Review authors 
completed these tasks independently and met to compare 
their results and reach consensus. 

Every selected study was allocated to a pair of review au- 
thors, with 10 review authors participating in total. Each 
review author independently extracted data from the in- 
cluded studies using the structured data extraction sheet 
shown in Online Supplementary Document. 

Each pair of reviewers compared their completed data ex- 
traction forms and any discrepancies between review au- 
thors' results were resolved through discussion; if no agree- 
ment could be reached, a third review author acted as an 



12208 records identified from database 
searching 



3117 duplicates excluded 



9091 records after removal of duplicates 



8780 records excluded after 
of title and abstract 



arbiter. Because 10 review authors participated in the data 
extraction process, some categories were interpreted differ- 
ently by some reviewers. Therefore, 3 reviewers went over 
the entire data extraction again to ensure uniformity. 

We contacted authors of studies containing incomplete 
data to request the missing information. Some authors did 
not reply to our request for additional information, whilst 
other authors did not know the answer to our questions. 
For 1 study the response obtained from the author resulted 
in the subsequent exclusion of the study from the system- 
atic review. 

Assessment of risk of bias in included 
studies 

During the data extraction process, we assessed the risk of 
bias at the outcome level using tools recommended by the 
Cochrane Collaboration [23]. For RCTs, we did so across 
the domains of (1) random sequence generation, (2) allo- 
cation concealment, (3) blinding of participants and per- 
sonnel, (4) blinding of outcome assessment, (5) incomplete 
outcome data, (6) selective outcome reporting, and (7) oth- 
er bias including the comparability of intervention and 
control group; characteristics at baseline; validation of out- 
come assessment tools; reliability of outcome measures; 
and protection against contamination. 

We assessed the risk of bias for cRCTs across 
the domains of (1) recruitment bias, (2) base- 
line imbalances, (3) loss of clusters and (4) in- 
correct analysis. For each study, 2 reviewers 
independently categorised each domain as low, 
high or unclear risk of bias. 



screening 



309 records included for full text reading 



102 records excluded after full text 
reading 



207 eligible articles were included 



59 classified as offline eLearning 
78 classified as simulation based 
eLearning 



70 eligible articles were included 



59 articles (60 trials) had relevant 
interventions and outcomes and were 
included in the analysis 



11 articles excluded from online 
eLearning. 

Reasons for exclusion: 

- Three studies awaiting author 
response, insufficient data for 
analysis 

- Two had no comparison group 

- One had participants that were not 
undergraduate students 

- One had an intervention that was 
not eLearning 

- One had a duplicate publication 

- Three had a design that was not 
RCTor cluster RCT 



Figure 1 . Flowchart of the studies included in the review. 



Summarising the data 

We qualitatively compared the characteristics 
of the participants and of the interventions be- 
tween the included studies to determine the 
feasibility of conducting a meta-analysis. Be- 
cause of substantial clinical, educational, con- 
tent and methodological heterogeneity we did 
not conduct a meta-analysis. Instead, we ad- 
opted a thematic summary approach [27]. 



RESULTS 

12 208 reports were identified from database 
screening of which 309 were retrieved for full- 
text evaluation of which 207 studies met the 
inclusion criteria (Figure 1). Fifty— nine arti- 
cles [25-82] complied with the term internet 
and local area network or online eLearning 
(see Online Supplementary Document for 
detailed description) and were included in the 
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analysis. One study [83] involved students in 2 consecu- 
tive RCTs that were analysed separately (Ainsworth 2012A, 
Ainsworth 2012B). Thus the total number of evaluated tri- 
als was 60. 

Included studies 

All studies were published in peer reviewed journals be- 
tween 2000 and 2013 except 1 dissertation [81]. All in- 
cluded studies were parallel RCTs or cRCTs. The included 
number of RCTs and cRCTs suggests an increase in the 
number of publications after 2007 as eighteen of the in- 
cluded studies (30%) have been published between 2000 
up and 2007 (ie, 8 years). The remaining 42 studies (70%) 
have been published in the shorter time interval between 
2008 and mid 2013 (ie, 5.5 years). Out of all 60 included 
studies, 33 studies investigated eLearning in the field of 
medicine [26,28,32,34-36,38-41,44,45,49,50,54,58- 
62,64-70,72,75,77,78,80,82]. Eleven of the articles [25,2 
7,33,42,43,53,56,74,76,79,83] were exclusively from 
nursing, 3 [55,57,81] were within the field of physical ther- 
apy, whereas 3 other studies within pharmacy [30,31,71]. 
Nine studies [29,37,46-48,51,63,73] investigated eLearn- 
ing for dentistry students. Additionally, 1 article [52] dealt 
with medicine, nursing, and physical therapy while the re- 
maining study recruited university students, but did not 
define their discipline [81]. 

Participant characteristics 

The total number of participants included across all trials 
was 6750 participants. The study with the smallest control 
group had 10 participants [77] whereas the largest control 
group had 249 participants [55] . The study with the small- 



est intervention group had 10 participants [33], while the 
largest intervention group had 349 participants [55] . Most 
studies were conducted among undergraduate university 
students apart from 9 studies [27,29-31,33,42,53,54,79] 
that investigated the effect of network— based eLearning for 
vocational training. Out of the 20 (33%) studies 
[30,32,33,40,45,47-49,52,53,55,57,64,65,71,73,75, 
76,83] that specified the age of the students, the lowest 
mean age of participants in a control group was 20.0 years 
[55] and the oldest was 30.0 years [76]. The lowest mean 
age in an intervention group was 19.9 years [75] and the 
highest was 30.0 years [76]. 

Intervention characteristics 

Fifty studies compared eLearning to traditional learning, 
and 10 studies [26,35,36,45,46,61-63,66,74] compared 
1 mode of eLearning to another mode of eLearning. The 
duration of exposure ranged from 9.05 minutes to 9 
months [83] . Most of the studies (5 1 out of 60,85%) were 
conducted exclusively in high income countries. Seven 
studies were conducted solely in low- to middle-income 
countries: 2 in Brazil [25,26] ; 2 in China [38,56] ; 1 in Thai- 
land [33] ; and 2 in the Chinese Taipei [27,42] . One study 
[52] was conducted simultaneously in Brazil and the USA 
(Figure 2). 

The majority of the studies used a website to present the 
learning material to the participants as part of their inter- 
vention [25-34,37-46,48-66,68-80,82,83]. Three studies 
[35,36,47] used a spaced education intervention, ie, an in- 
tervention in which the educational exposures are spaced 
and repeated over time [35]. In these studies the learning 
material was presented via email on a regular basis [35,36, 




Figure 2. Country of origin of included, examined studies for low- and middle-in- 
come and high income countries separately. 
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47] . One study used video conference lectures as an inter- 
vention [67] , and 1 study used a visual concept map [81] . 

Primary outcomes 

Students' knowledge assessment. The knowledge gained 
from the exposure to the intervention was assessed in a 
number of different ways in the included studies. Overall, 
53 [25-27,29-33,35-46,48-52,54,56-67,69,70,72-83] 
out of the 60 studies looked at a knowledge as an outcome. 
Nineteen of these studies [25,32,35,40,43,45,48-50,56, 
57,60,69,72,74,75,77,80,82] used only multiple choice 
questions (MCQs) to test students' knowledge and under- 
standing. Six studies used MCQs as a knowledge assess- 
ment tool together with an adaptive spaced test [36] , gap 
text questions [51], matching and short answer questions 
[61] , open ended and true/false questions [64] , short essay 
questions [81] and a key features test [59]. 

Seven studies reported using test questions [3 1 ,39,40,44,67, 
70,78] to assess knowledge of study participants. Six other 
studies used open ended [30,33,63,66] or Likert type ques- 
tionnaires [29] or even "fill in the blank" questions [73]. 
The rest of the studies measured students' knowledge gain 
via other testing means including general numeracy tests 
[83] ; written exams [41 ,46,58,65] ; independent observers' 
assessment [79]; cognitive assessment instruments [54]; 
surgical knowledge test scores [76] ; a Diagnostic Thinking 
Inventory and individual students' performance in solving 
clinical reasoning problems [38] ; a modified version of the 
Dartmouth Sleep Knowledge and Attitudes survey [62] ; an 
interactive evaluation about melanoma [26] ; an orthodon- 
tic examination form for each patient [37]; or some form 
of a knowledge assessment scale or checklist [42,52,68]. 

Students' skills assessment. Skills were evaluated in 16 
studies [28-30,32-34,39,42,53,55,57,63,68,69,71,72] 
using various methods to assess the outcome. Nine studies 
[33,34,42,55,63,68,71,72] used a rating scale and/or 
checklists (eg, an OSCE) to assess clinical skills. One study 
[53] used a search skills test, another 1 [29] a Likert type 
questionnaire while 3 studies evaluated students skills 
through written assessments such as data collection sheets 
[30] , written case analysis [41] and open questions on stan- 
dardised tasks [32]. Finally, 1 study [28] measured the de- 
gree of new skills acquisition by using a self-assessment 
report whilst another [39] measured the time that students 
made to complete the assigned exercise. 

Students' satisfaction and attitudes towards eLearning. 

Feedback from students assessed as their attitude towards 
the eLearning intervention was reported as an outcome in 
a total of 14 studies [28,29,32,33,43,45,48,49,54-56,63, 
64,72] . In all of these, students' attitude was measured by 
questionnaires. 



Student satisfaction was considered as an outcome in 33 
studies [25,32-34,37-41,43,46-48,50-52,55,57-61,64- 
66,68,69,73-76,80,82]. Seventeen of these studies 
[33,34,38,40,51,52,55,57,59,61,64-66,68,73,75,76] 
mentioned that student satisfaction was evaluated with Lik- 
ert scale questionnaires. The 16 remaining studies compar- 
ing student satisfaction among the students [25,32,37,39, 
41,43,46-48,50,58,60,69,74,80,82] used different types 
of questionnaires or surveys without mentioning the use of 
Likert scales. 

Secondary outcomes 

Cost-effectiveness of the eLearning interventions. 

Cost— effectiveness or cost-benefit or cost-utility of eLearn- 
ing interventions were not assessed in any of the studies, 
however, some of the studies mentioned several financial 
and resource related elements of eLearning. 

Buzzell et al. [54] mentioned that in the future many ex- 
perts would be involved in content generation for their re- 
spective disciplines and that content could be shared online 
among their disciplines. Thus the online content develop- 
ment and delivery would not need the involvement of 
many faculty at all stages of content development and in 
turn educational institutions would be cost efficient. Stain 
et al. [67] mentioned that the costs of setting up videocon- 
ferencing were comparable to the costs of live lectures after 
an initial hardware investment of less than US$10000. 
Stewart et al. [68] cited a paper saying that reduction of 
instructor training time, labour costs and institutional in- 
frastructure could result in significant cost-efficiency Tou- 
mas et al. [71] mentioned in the discussion that using the 
Internet leads to "reduced costs in terms of tutor-led work- 
shops and is more efficient, enabling more complex topics 
to be covered in workshops". Hauer et al. [34] deduced 
that the video cases were cheaper than the mini-CPX (Clin- 
ical Performance Evaluation) examination they used. An 
in-person examination of a class of 150 students cost ap- 
proximately US$ 5400, which did not yet include Clinical 
Skills Centre maintenance costs, costs of case development 
and payment of Centre staff. In contrast, plain technologies 
as video cases were produced at a total cost of US$ 2200. 
Besides, the video cases could be reused freely, whereas the 
in— person mini-CPX requires annual purchase of a license. 

In contrast, Fleming et al. [73] mentioned that the devel- 
opment of web-based or Computer Assisted Instruction is 
expensive in terms of time and energy spent. Phadtare et 
al. [52] made a general comment on the potential lack of 
necessary infrastructure and "new" costs associated with 
online courses. 

Adverse and/or unintended effects of eLearning. Ad- 
verse or unintended events of the eLearning intervention 
were not reported in any of the studies. 
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Excluded studies 

Initially, 65 studies were categorised as online eLearning 
studies. We reclassified 2 studies [84,85] as non-net- 
worked computer-based because their eLearning interven- 
tions could be fully functional even without network tech- 
nologies' support. Lhree studies [86-88] were excluded 
because of insufficient data while another [35] was exclud- 
ed as a duplicate paper. Seven studies [89-95] were ex- 
cluded during the data extraction process, just before the 
analysis, because they met 1 or more of the exclusion cri- 
teria. Four of these 7 studies [91,92,94,95] were excluded 
because their study design was not a parallel or cRCT eg, 
a cross-over design [94] . Two studies [89,93] were exclud- 
ed because they did not include comparison groups for the 
eLearning intervention eg, 2 different blended teaching 
methods using a common eLearning intervention in ex- 
actly the same way [93] . Finally, 1 study [90] used an eL- 
earning intervention which was considered ineligible for 
our study ie, electronic voting during the lecture [90]. 

Risk of bias in the included parallel RCTs 

Thirty-one of the studies were considered to be at high risk 
of bias. [25,33,34,36,38-41,44,49-51,53,55, 57,60,62- 
64,69,70,72,74-77,79-81,83]. Twenty-nine of the studies 
[26,30,32,35,43,48,52,54,56,59,61,65-67,73,78] had 1 
or more categories classified as an unclear risk of bias, es- 
pecially regarding the allocation of participants to interven- 
tion groups. There was only 1 study [47] with all the cat- 
egories classified as low risk of bias (Figure 3 and 4). Risk 
of bias is described in detail in the Online Supplementary 
Document. 

Effects of online eLearning interventions 

The 60 randomized trials included in our review assessed 
the effectiveness of online eLearning interventions in terms 
of knowledge, skills, attitudes and satisfaction. The find- 
ings were based on comparisons between online eLearning 
and traditional learning or between various modes of on- 
line eLearning. A study may have compared more than 1 

Random sequence generation (selection bias) 

Allocation concealment (selection bias) | 
Blinding of participants and personnel (performance bias) | 
Blinding of outcome assessment (detection bias) | 
Incomplete outcome data (attrition bias) | 
Selective reporting (reporting bias) 
Other bias 



Figure 4. Risk of bias for each 
individual parallel randomised 
controlled trial (RCT) 
separately. 




100% 



Low risk of bias 



^Unclear risk of bias 



High risk of bias 



Figure 3. Overall risk of bias graph. 
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outcome between groups, and each outcome may have 
been assessed in multiple ways. For example, a study which 
compared students' acquisition of skills may have assessed 
skills in terms of the student's performance on a global rat- 
ing scale, the ability to perform a specific procedure, as well 
as the ability to comply with the requirements in a check- 
list. As a result, the number of comparisons made across 
studies for a particular outcome may exceed the number 
of studies which reported on that outcome. Only 2 studies 
[32,33] measured all specified outcomes of knowledge, 
skills, attitudes and satisfaction. 

The studies were split into 2 research themes evaluating the 
impact of eLearning interventions for undergraduate health 
care education: traditional learning vs online eLearning, 
and online eLearning vs online eLearning. 

Traditional learning vs online eLearning 

Fifty of the 60 included studies (83%) compared online 
eLearning with traditional learning. [25,27—34,37-44,47- 
58,60,64,65,67-73,75-83]. Table 2 presents the summa- 
ry of the findings of the individual studies. Further descrip- 
tion of the nature of the interventions is in the Online 
Supplementary Document. 

Students' knowledge. Amongst the 60 studies which com- 
pared online eLearning with traditional learning, knowl- 
edge was assessed in 43 RCT studies (86%) [25,27,29— 
31,37-44,48-52,54,56-60,64,65,67,69,70,72,73,75-83] 
and 7 cRCT studies [27-29,31,42,58,71]. 

Twelve studies (27%) assessing knowledge gain demon- 
strated significantly higher knowledge gains for students 
assigned to the online eLearning compared to those ex- 
posed to traditional learning [31,41,49,51,52,56,60,70,72, 
77,79,83] . Outcome measures for these studies were based 
on test items or questions [31,70], written case analyses 
[41], MCQs [50,51,56,60,72,77], the Six-subgroup Qual- 
ity Scale (SSQS) [52] , a general numeracy test [83] and in- 
dependent assessments by evaluators [79] . The sample size 
of these studies ranged from 39 to 1475. Six of these 12 
studies were conducted on medical students [41 ,49,60,70, 
72,77], 3 among nursing students [56,79,83], 1 among 
dentistry students [51], 1 among pharmacy students [31], 
while 1 study [52] was conducted among medicine, nurs- 
ing, and physical therapy students. Five of these studies 
used full online eLearning as the main intervention [49,52, 
70,77,79] whereas 7 used blended learning [31,41,51,56,60, 
72,83]. 

Post— intervention knowledge was not significantly differ- 
ent between eLearning and traditional learning in 24 (48%) 
of the included studies [25,32,33,37,38,42-44,48,50,54, 
57-59,64,65,69,73,75,76,78,81,83]. Three studies [27,30, 
39] showed mixed results ie, favouring the intervention, 
control or neither 1 depending on the specific indicator of 



knowledge being assessed. In 1 (2%) study [82] knowledge 
was assessed but not tested for statistically significant dif- 
ferences between the intervention groups. 

Finally, there were 3 studies [29,30,80] that demonstrated 
significantly higher knowledge gains for students assigned 
to traditional eLearning compared to those exposed to on- 
line eLearning. Two of these studies [29,80] used full on- 
line eLearning as the main intervention while the other 1 
[40] used blended learning. 

Students' skills. Overall, 15 studies - 11 RCTs [30,32- 
34,39,53,55,57,68,69,72] and 4 cRCTs [28,29,42,71] 
measured skills as outcome. 

Of the studies that evaluated differences in skill acquisition, 
6 (40%) found significantly greater skill acquisition amongst 
students assigned to online eLearning [33,34,42,57,71,72] . 
The number of participants included in these studies ranged 
from 44 to 303. Two of these studies were conducted in 
medical students [34,72], 2 in nursing students [33,42], 1 
in physiotherapy students [57] and 1 in pharmacy students 
[71]. Four of these studies used traditional learning as their 
main intervention [34,42,57,71], whereas 2 used blended 
learning as the main intervention [33,72] . 

Three studies (2 1 %) did not detect a significant difference 
in skill acquisition between groups [39,53,55] . One study 
[30] showed mixed results ie, favouring the online eLearn- 
ing or the traditional learning group depending on the spe- 
cific indicator of skills being assessed. This study had 3 
groups, comparing pharmacy students' knowledge and 
ability to assess metered-dose inhaler (MDI) after a lecture 
based tutorial, a web-based tutorial and being provided no 
teaching on the topic at all. Finally, there was 1 study [69] 
that demonstrated significantly higher skill gains for stu- 
dents assigned to traditional learning compared to those 
exposed to online eLearning. This study used full eLearn- 
ing as the main intervention. 

Students' attitudes and satisfaction. Twelve studies 
(24%) - 10 RCTs [32,33,43,48,49,54-56,64,69,72] and 2 
cRCTs [28,29] — assessed attitudes as an outcome of the 
intervention through questionnaires. 

Eight of these studies [29,32,33,43,49,50,54,56] (67%) 
did not find a statistically significant difference between the 
2 types of learning methods, or the study showed mixed 
results for online eLearning vs traditional learning depend- 
ing on the test evaluated. Three studies [28,55,72] assessed 
attitude, but did not test for statistically significant differ- 
ences between the intervention groups. None of the stud- 
ies reported a significant result on student attitudes favour- 
ing online eLearning interventions. 

The remaining study (8%) [64] reported more positive at- 
titudes towards the intervention in the traditional learning 
groups. This study used full online eLearning as the main 
intervention. 
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Table 2. Summary of findings from the 50 studies which compared online eLearning with traditional learning 


Study 


Discipline 


Knowledge Skills 


Attitude Satisfac- 


No. OF 


Intervention delivery approach 


Characteristics 










tion 


PARTICIPANTS 






Ainsworth 2011a [83] 


Nursing 


NS 












Ainsworth 2011b [83] Nursing 


E 












Arroyo— Morales 2012 


Physiotherapy NS 


E 


NS 


46 


Blended learning 


IG: Online tutorial (ECOFISIO) 


[57] 














CG: Self— study 


Baumlin 2000 [58] 


Iviedicme 


NS 




NS 


100 


Blended learning 


IG: Computer Assisted Instructions 

(EMCyber-School) 

CG: Lectures 


Beeckman 2007 [79] 


Nursing 


E 






426 


Full eLearning 


IG: 1 h eLearning program 
CG: 1 h lecture 


Brettle 2013 [53] 


Nursing 




NS 




77 


Full eLearning 


IG: Online tutorial 

CG: 1 h face— to— face tutorial 


Buzzell 2002 [541 


Medicine 


NS 




NS 


34 


IG1: Full eLearning 
IG2: Blended learning 


IG: Web-based tutorials 
IG2: Traditional lectures with web- 
based tutorials 
CG: Traditional lectures 


Cantarero2012 [55] 


Physiotherapy 




NS 


DNT DNT 


50 


Full eLearning 


IG: Online materials 

CG: access to books and documents 


Chen 2007 [81] 


Undefined, 


IN j 






145 


IG1: Full eLearning 
IG2: Traditional learning 


IGI: A visual advance organizer 
IG2: Text outline 
CG: Textbook reading without an 
advance organizer (AO) 


Chen 2012 [27] 


Nursing 


M 






146 


Blended learning 


IG: Online tests 

CG: Group A: Paper references, 

Group B: No assistance 


Cox 2008 [28] 


^Medicine 




DNT 


DNT 


138 


IG1: Traditional learning 
IG2: Blended learning 


IGI: Group discussion 

IG2: Website, Videos 

CG: Paper based, traditional learning 

materials 


DeBate 2013 [29] 


Dentistry 


T 


DNT 


NS 


608 


Full eLearning 


IG: Online (computer and website) 
CG: Regular curriculum 


Erickson 2003 [30] 


Pharmacy 


M 


M 




42 


IG1: Traditional learning 
IG2: Full eLearning 


IGI: Lecture— based tutorial 
IG2: Web-based tutorial 
CG: No intervention 


Fernandez 2011 [76] 


Nursing 


N^ 
IN J 




NS 


116 


Full eLearning 


IG: Computer assisted learning 
CG: Face— to-Face lecture 


Fleming 2003 [73] 


Dentistry 


NS 




NS 


31 


eLearning and traditional IG: Slide/audiotape self-instruction 














learning separately 


and web-based self-instruction 
CG: Web— based self— instruction and 
slide/audiotape self— instruction 


Flowers 2010 [31] 


Pharmacy 


E 






79 


Blended learning 


IG: Web-based Multimedia Vignettes 
CG: No Web-based Multimedia 
Vignettes 


Friedl 2006 [32] 


Medicine 


NS 


DNT 


NS NS 


126 


Full eLearning 


IG: online multimedia course 
CG: Text books 


Gerdprasert 2010 [33] Nursing 


NS 


E 


NS NS 


84 


Blended eLearning 


IG: Web, interactive graphics, 
















animation 
















CG: Traditional teaching 


Hauer 2013 [341 


Medicine 




E 


E 


303 


IG1: Full eLearning 
IG2: Traditional learning 


IG: Web based module 
CG: Group work, role play. 


Jenkins 2008 [78] 


Medicine 


NS 






73 


Blended learning 


IG: Internet— based tutorial 
CG: Lecture. 


Juliani2011[25] 


Nursing 


NS 




DNT 


80 


Full eLearning 


IG: Designing a schedule with 
internet 

CG: Designing a schedule without 
internet 


Kandasamy 2009 [80] 


Medicine 


L 




NS 


62 


Full eLearning 


IG: Online CAI module 

CG: Review articles (Text based) 


Komolpis 2002 [37] 


Dentistry 


NS 




NS 


99 


Full eLearning 


IG: Digital records on PC 
CG: Hardcopy records 


Lee 2010 [38] 


Medicine 


NS 




NS 


52 


Blended learning 


IG: Web-based problems, workshop 
CG: No workshop 


Leong 2003 [39] 


Medicine 


M 


NS 


NS 


54 vs 
325* 


IGI: Full eLearning 
IG2: Full eLearning 


IGI: Computer based cases (other 
than C1+C2) 

IG2: Computer based cases (C1+C2) 
CG: No Computer based cases 
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Table 2. continued 


Study 


Discipline 


Knowledge Skills 


Attitude Satisfac- 
tion 


No. OF 

PARTICIPANTS 


Intervention delivery approach 


Characteristics 


Lewis 2011 [40] 


Medicine 


T 


NS 


39 


Blended learning 


IG: Web-based MCQ's 
CG: Textbook resource 


Lipman 2001 [41] 


Medicine 


E 


NS 


130 


Blended learning 


IG: Website, books 
CG: Books, discussions 


Lu 2009 [42] 


Nursing 


NS E 




147 


Blended learning 


IG: Lectures and interactive web- 
based course 
CG: lectures only 



Maag 2004143] Nursing NS NS NS 96 IG1: Traditional learning IG1: Text and image 

IG2: Blended learning IG2: Text and image and animation 
IG3: Blended learning IG3: Text, Image, Animation, and 
Interactivity 
CG: Text modules 



Mahnekn 2010[44] Medicine NS 96 IG1: Blended learning IG1: eLearning, self 

IG2: Blended learning IG2: eLearning, mandatory 
CG: No access to eLearning 



Nkenke 2012 [48] 


Dentistry 


NS 




NS 


NS 


42 


Blended learning 


IG: Technology enhanced learning 
CG: Didactic lectures, PowerPoint 
presentation 


Nkenke 20 12 [47] 


Dentistry 








NS 


42 


Blended learning 


IG: Spaced education 
CG: Lectures 


Ochoa 2008 [49] 


Medicine 


E 




NS 






Full eLearning 


IG: Web— based interactive program 
CG: Traditional text. 


Palmer 2008 [50] 


Medicine 


NS 






DNT 


130 


IG1: Traditional learning 
IG2: Blended learning 
IG3: Blended learning 


IG1: Written case— studies 
IG2: Clinical material + interactive 
computer— based case studies 
IG3: Clinical material + interactive 
computer— based case studies 


Peroz 2009 [51] 


Dentistry 


E 






NS 


85 


Blended learning 


IG: Online education 

CG: PowerPoint, discussions 


Phadtare2009 [52] 


Medicine; 

Nursing; 

Physiotherapy 


E 






E 


48 


Full eLearning 


IG: Online materials 
CG: Off-line materials 


Raupach 2009 [59] 


Medicine 


NS 






DNT 


148 


Blended learning 


IG: Web— based teaching module 
CG: lace— to— lace traditional lecture 


Raupach 2010 [60] 


Medicine 


E 






E 


74 


Blended learning 


IG: Web— based module 
CG: Traditional lecture 


Ricks 2008 [77] 


Medicine 


E 








23 


Full eLearning 


IG: Computer Assisted Learning 
group 

CG: No intervention 


Smits 2012 [64] 


Medicine 


NS 




T 


NS 


128 


Full eLearning 


IG: Case based eLearning 
CG: Text based learning 


Spikard 2002 [65] 


Medicine 


NS 






E 


95 


Full eLearning 


IG: Online lecture 

CG: Traditional learning 


Stain 2005 [67] 


Medicine 


NS 








12 vs 98* 


Full eLearning 


IG: Videoconference lectures 
CG: Conventional lectures 


Stewart 2013 [68] 


Medicine 




DNT 




NS 


71 


Blended learning 


IG: Online access to learning content 
CG: Standard content 


Stolz 2012 [69] 


Medicine 


NS 


T 




T 


129 


Full eLearning 


IG: Web— based training 
CG: Lectures 


Subramanian 2012 
[70] 


Medicine 


E 








33 


Full eLearning 


IG: Interactive medical soltware 
CG: Lectures 


Succar2010 [82] 


Medicine 


DNT 






NS 


147 


Blended learning 


IG: Computer based training 
CG: Traditional teaching 


Toumas 2009 [71] 


Pharmacy 




E 






236 


Blended learning 


IG: Internet— based Tutorial 
CG: Small group workshop 


Truncali 2011 [72] 


Medicine 


E 


E 


DNT 




141 


Blended learning 


IG: Web— based Tutorial 
CG: Lectures 


Wang 2009 [56] 


Nursing 


E 




NS 




133 


Blended learning 


IG: Online, sell— learning 

CG: Traditional multimedia lecture 


Yeung2013 [75] 


Medicine 


NS 






NS 


78 


Blended learning 


IG: Computer— assisted learning 
CG: Text— /image— based learning 
(TBL) 



E — Results lavoured online eLearning over traditional learning, NS — No significant difference between online eLearning and traditional learning, MCQ 
— Multiple choice questions M — Mixed results, T — Results lavoured traditional learning over online eLearning, DNT — Difference not tested, CG — Con- 
trol group, IG — Intervention group 

* Average number students exposed to videoconlerence lectures and conventional lectures, respectively. 
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Student satisfaction was assessed in 28 RCTs [25,32— 
34,37-41,43,47,48,50-52,55,57,59,60,64,65,68,69,73, 
75,76,80,82] and 1 cRCT [58]. 

Out of 29 studies looking at the level of student satisfaction, 
4 [34,52,60,65] (14%) found a significantly greater propor- 
tion of students exposed to online eLearning who were sat- 
isfied compared to those exposed to traditional learning. 
One of these 4 studies [60] compared blended learning with 
traditional learning, whilst the other 3 [34,52,65] used full 
eLearning interventions compared with traditional learning 
ones. Twenty studies (74%) did not detect any significant 
difference [32,33,37-41,43,47,48,51,57,58,64,68,74- 
76,80,82] while in 4 studies satisfaction was assessed 
[25,50,55,59] but not tested for statistically significant dif- 
ferences between the intervention groups. 

There was 1 study [69] using full online eLearning as the 
main intervention that reported a statistically significant 
better student satisfaction in the traditional learning group. 

Comparison of different types of eLearning 
against each other 

Ten (18%) of the included studies [26,35,36,45,46,61- 
63,66,74] compared the effectiveness of various modes of 
online eLearning against each other. Eight of these studies 
[26,35,45,46,61-63,74] compared groups of eLearning 
with different levels of student interaction. In 2 of them 
"interactivity" was also facilitated by collaborative tools, ie, 
online web chats [74] and discussion forums and online 
message systems [61]. 

Students' knowledge. All of the 10 studies [26,35,36,45,46, 
61—63,66,74] comparing various forms of online eLearning 
measured and reported their effects on knowledge. 

Five studies observed a difference in results between dif- 
ferent modalities of eLearning. In a study comparing an 
adaptive form of spaced education against a linear, repeti- 
tive one [36] , the adaptive eLearning intervention showed 
better results than its "passive" form. Another study show- 
ing significant knowledge acquisition for an "active" eL- 
earning intervention was Chao et al [26] where a linear 
educational environment (website) supported by comple- 
mentary information (skin anatomy images) which users 
could access at will was compared to a non modified web- 
site. Similarly, in 1 study [63] an eLearning intervention, 
allowing students to play a video back and forth at their 
will showed better knowledge gains in comparison to an 
eLearning intervention where the procedure was linear. In 
a study [61] on a "passive" type of eLearning, offering 
course material through conventional World Wide Web 
technology and by letting students engage with the instruc- 
tor only by email resulted in higher knowledge gains in 
comparison to an interactive eLearning intervention where 



students could make use of all the learning tools of the Web 
CT (online proprietary virtual learning environment sys- 
tem) [61]. A "passive" eLearning intervention showed fa- 
vourable results also for Salas et al [62]. In this study, par- 
ticipants in the "passive" eLearning group were solely 
provided with a list of random sleep facts and trivia pre- 
sented in a PowerPoint format. The "active" eLearning in- 
tervention consisted of an online, self— paced, sleep medi- 
cine learning module. 

Non statistical significant differences were found in 4 stud- 
ies [35,45,66,74] comparing different online eLearning 
modalities. One study showed no difference in knowledge 
acquisition between eLearning modes [46] . 

Students' skills. Skill acquisition was assessed in 1 study 
[63] . This study showed no significant differences in skills 
acquisition between the 2 different (active vs passive) eL- 
earning modalities. 

Students' attitudes and satisfaction. Manikam et al. [45] 
and Schittek Janda et al. [63] were the only studies amongst 
the 10 studies comparing different eLearning modalities that 
assessed attitude. The study by Manikam et al showed no 
difference in students' attitudes between the 2 eLearning 
modes. In this study a dummy learning package was com- 
pared to the ABD learning package, ie, symptom-based de- 
cision-making pathways software. Schittek Janda et al. re- 
ported no significant differences in skills acquisition between 
the 2 different (active vs passive) eLearning modalities. 

Four (40%) studies [46,61,66,74] compared the effects of 
different eLearning modes on student satisfaction. Two 
studies [46,61] showed no difference in students' satisfac- 
tion for the 2 eLearning modes. Frith et al. [74] reported 
that students in the group that used collaboratively a 6- 
week Web— based course on cardiac rhythm interpretation 
supported by online chat software were more satisfied than 
students in the group who worked on the same course in- 
dependently. In the study by Spickard et al. [66] students 
in the groups of the online lecture of power point slide pre- 
sentation with audio narration were more satisfied than 
students in the group of the online lecture of power point 
slide presentation without audio narration. 

DISCUSSION 

Summary of main results 

This systematic review reports on the effectiveness of on- 
line eLearning for undergraduates in health professions. 
We found that online eLearning does lead to changes in 
knowledge, skills, attitude and satisfaction and seems to be 
more effective than traditional learning in terms of knowl- 
edge and skills gained. Our findings are similar to previous 
reviews of online eLearning [21,22,96—100] and offline eL- 
earning [18] . 
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In our review, 29% of the studies showed significantly 
higher knowledge gains, 40% of the studies showed sig- 
nificantly greater skill acquisition, 67% of the studies 
showed no difference in attitude and 14% of the studies 
showed higher satisfaction with online eLearning than tra- 
ditional learning. The participants in the included studies 
were from the fields of medicine, dentistry, pharmacy or 
medical allied studies enrolled at universities, with a small- 
er number conducted at vocational training centres or col- 
leges. Consequently, the results of this systematic review 
apply to students from similar disciplines, universities and 
colleges. The majority of the studies were conducted in 
high-income countries with exception of few [25,26,33, 
38,42,52,56,101] which were from low to middle income 
countries, hence these results are generalizable only to their 
corresponding settings. 

The studies included in our review had a high degree of 
methodological, educational and clinical heterogeneity, 
similar to previous reports [21,22,96-100] . Knowledge as- 
sessment was, for example, conducted using different test 
items or questions [31,70], written case analyses [41], 
MCQs [49,51,56,60,72,77] the Six-subgroup Quality 
Scale (SSQS) [52], a general numeracy test [83] and inde- 
pendent assessments. Similarly there was variability in the 
assessment of skills, attitudes and satisfaction across the 
studies. Hence pooling of effect estimates was not possible. 
Mode of online interventions varied across the studies, 
most of the studies used a website, while some used other 
interventions such as spaced education, video lectures or 
visual concept map. Furthermore, there were great varia- 
tions in exposure time to the eLearning intervention. Fi- 
nancial and resource related elements of eLearning was re- 
ported only in 8 studies [34,52,54,59,67,68,71,73]. 
Nevertheless, none of the studies included a robust cost- 
effectiveness analysis of eLearning vs traditional learning 
and therefore it is not possible to provide an assessment on 
cost— effectiveness of online eLearning. Furthermore, no 
studies reported on the adverse effects of online eLearning. 

The overall quality of evidence included in this systematic 
review is not uniform and contains a significant number of 
studies with methodological weaknesses with only 1 high 
quality study [48] ; similar findings were reported in previ- 
ous reviews. [16,19,20,22,102,103] Most of the included 
studies did not adhere to the CONSORT guidelines for re- 
porting of RCTs [104] and thus their risk of bias was un- 
clear. Several of the included studies had high risk of bias 
due to volunteer [49,51,57,60,72,74,83] and attrition bias. 
[34,41,70,72,79]. Due to the weaknesses of most of the 
included studies a strong conclusion on whether there is a 
clear difference between online eLearning and traditional 
learning effectiveness that applies to the general population 
of learners cannot be drawn. 



Our study has many strengths. The review was based on a 
thorough search of available literature which identified a 
large number of potentially eligible studies identified and 
synthesized by a multi-disciplinary international team and 
it offers a number of advantages over previous work in this 
area. The key strength is an attempt to combine breadth of 
scope in terms of widely defining eLearning and the range 
of health professions covered. The review encompasses all 
empirical studies (RCTs). To ensure data quality, article 
screening and data extraction was done independently by 
2 persons to avoid subjective bias, disagreements were re- 
solved through discussion. The review included studies 
from both developed and developing countries and thus 
provides crucial information on the usage, effectiveness and 
applicability of online eLearning in these settings. Finally, 
the review used standard methods for systematic reviews 
and meta-analyses in accordance with preferred reporting 
items for systematic reviews and meta-analyses (PRISMA) 
which makes it transparent. 

The review had a few limitations. The included studies had 
several methodological rudiments, we contacted the au- 
thors to obtain necessary information for assessing the risk 
of bias for these studies, however due to time constraints, 
and it was not possible to contact all authors. Moreover, 
due to the lack of a uniform, standardized terminology for 
eLearning studies, we categorized studies as online eLearn- 
ing (ie, local area network or web— based) and offline eL- 
earning (ie, non— networked or computer based). Although 
we assigned each individual study to only 1 category, it is 
important to highlight that there might be some degree of 
overlap between categories as 1 form of technology may be 
built on another one. 

In summary, this systematic review compares online eL- 
earning and traditional learning in undergraduate health- 
related students and consolidates current knowledge on 
the effectiveness of online eLearning. The evidence from 
the highest and the lowest quality studies indicates that on- 
line eLearning is equivalent to and perhaps even more ef- 
fective than traditional learning in terms of knowledge and 
skills gained. The generalization of these findings is limited 
only to the studied population in the review. 

Online learnings ubiquity provides a convenient and pos- 
sibly a more cost-effective alternative to traditional learn- 
ing and has great potential in supporting health care work- 
force capacity building and competency development 
globally. This review highlights the need for improvements 
in the methodological design in future studies. 

Implications for policy makers 

The findings of this review present a potential incentive for 
policy makers to encourage adoption and the development 
of online eLearning programs. These online eLearning pro- 



June2014 • Vol. 4 No. 1 • 010406 



12 



www.jogh.org • doi: 10.7189/jogh.04.010406 



Online eLearning for undergraduates in health professions: A systematic review 



grams could be useful in training health care professionals 
in countries with acute health care worker shortage, with- 
out substantial investments. These online technologies if 
adopted earlier could help lower the burden of diseases by 
increasing the health care professional per capita. Though 
adoption of these online technologies would involve some 
initial start-up cost, it would be largely beneficial as the 
potential for the return on investment is high in terms of 
health gains and lives saved. 

Implications for educational institutions 

Online eLearning offers many opportunities. This review 
shows that eLearning is as effective as the traditional learn- 
ing and with many advantages compared to traditional 
learning. The universities could adopt these technologies 



and could reach out to a wider audience within and out- 
side their country, thus offering a tremendous growth op- 
portunity for the educational institutions. Institutions 
could employ online eLearning to train their health work- 
force without having to spend for their travel elsewhere 
within or outside their countries. 

Implications for future research 

The findings of the review have many implications for re- 
search. Future evaluations of online eLearning should aim 
to answer many remaining research questions from inter- 
vention design features to setting or modality for online 
eLearning, and build cost-effectiveness models. We should 
especially aim to strengthen the evidence base for develop- 
ing countries. 
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